
<html><head>
<title>flibs/strings - flibs</title>
<style type="text/css"><!--
    HTML {
	background: 	#FFFFFF;
	color: 		black;
    }
    BODY {
	background: 	#FFFFFF;
	color:	 	black;
    }
    DIV.doctools {
	margin-left:	10%;
	margin-right:	10%;
    }
    DIV.doctools H1,DIV.doctools H2 {
	margin-left:	-5%;
    }
    H1, H2, H3, H4 {
	margin-top: 	1em;
	font-family:	sans-serif;
	font-size:	large;
	color:		#005A9C;
	background: 	transparent;
	text-align:		left;
    }
    H1.title {
	text-align: center;
    }
    UL,OL {
	margin-right: 0em;
	margin-top: 3pt;
	margin-bottom: 3pt;
    }
    UL LI {
	list-style: disc;
    }
    OL LI {
	list-style: decimal;
    }
    DT {
	padding-top: 	1ex;
    }
    UL.toc,UL.toc UL, UL.toc UL UL {
	font:		normal 12pt/14pt sans-serif;
	list-style:	none;
    }
    LI.section, LI.subsection {
	list-style: 	none;
	margin-left: 	0em;
	text-indent:	0em;
	padding: 	0em;
    }
    PRE {
	display: 	block;
	font-family:	monospace;
	white-space:	pre;
	margin:		0%;
	padding-top:	0.5ex;
	padding-bottom:	0.5ex;
	padding-left:	1ex;
	padding-right:	1ex;
	width:		100%;
    }
    PRE.example {
	color: 		black;
	background: 	#f5dcb3;
	border:		1px solid black;
    }
    UL.requirements LI, UL.syntax LI {
	list-style: 	none;
	margin-left: 	0em;
	text-indent:	0em;
	padding:	0em;
    }
    DIV.synopsis {
	color: 		black;
	background: 	#80ffff;
	border:		1px solid black;
	font-family:	serif;
	margin-top: 	1em;
	margin-bottom: 	1em;
    }
    UL.syntax {
	margin-top: 	1em;
	border-top:	1px solid black;
    }
    UL.requirements {
	margin-bottom: 	1em;
	border-bottom:	1px solid black;
    }
--></style>
</head>
<! -- Generated from file 'strings/tokenize.man' by tcllib/doctools with format 'html'
   -->
<! -- Copyright &copy; 2008 Arjen Markus &lt;arjenmarkus@sourceforge.net&gt;
   -->
<! -- CVS: $Id: tokenize.html,v 1.2 2013/05/13 08:03:20 knystrom Exp $ flibs/strings.n
   -->
<body><div class="doctools">
<h1 class="title">flibs/strings(n) 1.1  &quot;flibs&quot;</h1>
<div id="name" class="section"><h2><a name="name">Name</a></h2>
<p>flibs/strings - Tokenizing strings</p>
</div>
<div id="toc" class="section"><h2><a name="toc">Table Of Contents</a></h2>
<ul class="toc">
<li class="section"><a href="#toc">Table Of Contents</a></li>
<li class="section"><a href="#synopsis">Synopsis</a></li>
<li class="section"><a href="#section1">Description</a></li>
<li class="section"><a href="#section2">INTERFACE</a></li>
<li class="section"><a href="#copyright">Copyright</a></li>
</ul>
</div>
<div id="synopsis" class="section"><h2><a name="synopsis">Synopsis</a></h2>
<div class="synopsis">
<ul class="syntax">
<li><a href="#1"><b class="cmd">use tokenize</b></a></li>
<li><a href="#2"><b class="cmd">call set_tokenizer( token, gaps, separators, delimiters )</b></a></li>
<li><a href="#3"><b class="cmd">part = first_token( token, string, length)</b></a></li>
<li><a href="#4"><b class="cmd">part = next_token( token, string, length)</b></a></li>
</ul>
</div>
</div>
<div id="section1" class="section"><h2><a name="section1">Description</a></h2>
<p>The <em>tokenize</em> module provides a method to split a string
into parts according to some simple rules:</p>
<ul class="itemized">
<li><p>A string can be split into &quot;words&quot; by considering spaces and commas and
such as separating the words. Two or more such characters are treated as
a single such separator. In the terminology of the module they represent
gaps of varying width. As a consequence there are no zero-length words.</p></li>
<li><p>A string can be split into &quot;words&quot; by considering each individual comma
as a separator. A string like &quot;One,,two&quot; would then be split into three
fields: &quot;One&quot;, an empty field and &quot;two&quot;.</p></li>
<li><p>Just like Fortran's list-directed input, the module handles strings with
delimiters: &quot;Just say 'Hello, world!'&quot; would be split in &quot;Just&quot;, &quot;say&quot;
and &quot;Hello, world!&quot;.</p></li>
</ul>
<p>The module is meant to help analyse input data where list-directed
input can not be used: for instance because the data are not separated
by the standard characters or you need a finer control over the handling
of the data.</p>
</div>
<div id="section2" class="section"><h2><a name="section2">INTERFACE</a></h2>
<p>The module contains three routines and it defines a single derived type
and a few convenient parameters.</p>
<p>The data type is <em>type(tokenizer)</em>, a derived type that holds all
the information needed for parsing the string. It is initialised via the
<em>set_tokenizer</em> subroutine and it is meant for the string passed to
the <em>first_token()</em> function. If you want to reuse it for a
different string, but the same definition, simply use
<em>first_token()</em> on the new string.</p>
<dl class="definitions">
<dt><a name="1"><b class="cmd">use tokenize</b></a></dt>
<dd><p>To import the definitions, use this module.</p></dd>
<dt><a name="2"><b class="cmd">call set_tokenizer( token, gaps, separators, delimiters )</b></a></dt>
<dd><p>Initialise the tokenizer &quot;token&quot; with various sets of characters
controlling the splitting process.
otherwise.</p>
<dl class="arguments">
<dt>type(tokenizer) <i class="arg">token</i></dt>
<dd><p>The tokenizer to be initialised</p></dd>
<dt>character(len=*) <i class="arg">gaps</i></dt>
<dd><p>The string of characters that are to be treated as &quot;gaps&quot;. They take
precedence over the &quot;separators&quot;. Use &quot;token_empty&quot; if there are none.</p></dd>
<dt>character(len=*) <i class="arg">separators</i></dt>
<dd><p>The string of characters that are to be treated as &quot;separators&quot;.
Use &quot;token_empty&quot; if there are none.</p></dd>
<dt>character(len=*) <i class="arg">delimiters</i></dt>
<dd><p>The string of characters that are to be treated as
&quot;delimiters&quot;. Use &quot;token_empty&quot; if there are none.</p></dd>
</dl></dd>
<dt><a name="3"><b class="cmd">part = first_token( token, string, length)</b></a></dt>
<dd><p>Find the first token of the string (also initialises the tokenisation
for this string). Returns a string of the same length as the original
one.</p>
<dl class="arguments">
<dt>type(tokenizer) <i class="arg">token</i></dt>
<dd><p>The tokenizer to be used</p></dd>
<dt>character(len=*) <i class="arg">string</i></dt>
<dd><p>The string to be split into tokens.</p></dd>
<dt>integer, intent(out) <i class="arg">length</i></dt>
<dd><p>The length of the token. If the length is -1, no token was found.</p></dd>
</dl></dd>
<dt><a name="4"><b class="cmd">part = next_token( token, string, length)</b></a></dt>
<dd><p>Find the first token of the string (also initialises the tokenisation
for this string). Returns a string of the same length as the original
one.</p>
<dl class="arguments">
<dt>type(tokenizer) <i class="arg">token</i></dt>
<dd><p>The tokenizer to be used</p></dd>
<dt>character(len=*) <i class="arg">string</i></dt>
<dd><p>The string to be split into tokens.</p></dd>
<dt>integer, intent(out) <i class="arg">length</i></dt>
<dd><p>The length of the token. If the length is -1, no token was found.</p></dd>
</dl></dd>
</dl>
<p>Convenient parameters:</p>
<ul class="itemized">
<li><p><em>token_whitespace</em> - whitespace (a single character)</p></li>
<li><p><em>token_tsv</em> - tab, useful for tab-separated values files</p></li>
<li><p><em>token_csv</em> - comma, useful for comma-separated values files</p></li>
<li><p><em>token_quotes</em> - single and double quotes, commonly used delimiters</p></li>
<li><p><em>token_empty</em> - empty string, useful to suppress any of the
arguments in the <em>set_tokenizer</em> routine.</p></li>
</ul>
</div>
<div id="copyright" class="section"><h2><a name="copyright">Copyright</a></h2>
<p>Copyright &copy; 2008 Arjen Markus &lt;arjenmarkus@sourceforge.net&gt;</p>
</div>
</div></body></html>